Finding "Patterns of Life" in NYC Taxi Traffic¶
In this notebook, we use tensor decompositions in order to find coherent "patterns of life" in NYC taxi trip records. We show that ENSIGN, when applied to spatiotemporal, entity-based data, can extract distinct travel, work, and leisure activities.
Table of Contents¶
# data manipulation
import numpy as np
import pandas as pd
# ENSIGN tools
from ensign.csv2tensor import csv2tensor
from ensign.cp_decomp import cp_apr, read_cp_decomp_dir, write_cp_decomp_dir
from ensign.visualize import plot_component
# custom plotting
import plotly.express as px
# Needed to display visuals in Jupyter Notebook
%matplotlib inline
Data ¶
We explore data provided by the NYC Taxi & Limousine Commision on taxi rides. The data includes features such as pickup and dropoff times and locations, the distance of the trip, the number of passengers, the payment amount and method, and more. As we are specifically interested in "patterns of life", or the types of trips and reasons for trips, we will focus on the trip times and locations. We consider just one week of data from June 13-19, 2016, but several years of data can be found and the T&LC site.
pd.read_csv('data/taxi_data.csv')
Tensor Construction ¶
When constructing a tensor for decomposition, the main considerations are which features to select and how to discretize them. This discretization process, known as binning, ensures that similar values in the chosen dimensions have the same tensor index and results in more coherent patterns. The relevant data here are the time of the trip and the starting and ending locations, so we select five columns: pickup time, pickup latitude, pickup longitude, dropoff latitude, and dropoff longitude. We round the times to the nearest hour and round the coordinates to three points of precision. Moreover, we fuse the latitude and longitude so that the indices in those modes represent specific locations. The spatial binning, together with the fusing operation, results in mode indices corresponding roughly to city blocks.
tensor = csv2tensor(
filepaths='data/taxi_data.csv',
columns=[
'tpep_pickup_datetime', 'pickup_longitude', 'pickup_latitude', 'dropoff_longitude', 'dropoff_latitude'
],
types=['datetime', 'float64', 'float64', 'float64', 'float64'],
binning=['hour', 'round=3', 'round=3', 'round=3', 'round=3'],
sort=['tpep_pickup_datetime'],
fuse_columns=[['pickup_longitude', 'pickup_latitude'], ['dropoff_longitude', 'dropoff_latitude']]
)
Tensor Decomposition ¶
We use CP-APR to decompose the tensor because we constructed the tensor to have count entries. A rank-100 decomposition extracted coherent patterns, and raising the rank did not yield further components of interest.
decomp = cp_apr(tensor, 100, mem_limit_gb=16)
write_cp_decomp_dir('taxi_decomposition', decomp, write_tensor=True)
Evaluating the decomposition quality: The CPDecomp object provides a dictionary metrics as a field that contains information on the decomposition: running time, various quality metrics, and the number of completed iterations. It is not necessary to get a perfect fit of 1 in order to have high-quality, interpretable components. Here, the fit on the order of 10-1 coupled with a high cosine similarity indicate good decomposition results. Therefore, we can be confident that the decomposition components capture almost all of the activity in the original data.
decomp.metrics
Component Visualization and Interpretation ¶
We can visualize each component by plotting the scores in each mode vector involved in the outer product reconstructing that component. The labels along each mode correspond to the binned values created during tensor construction. Any tuple of scoring indices in the outer product is a tensor index involved in the pattern described by the component. Therefore, the labels of the scoring indices describe the pattern. Specifically, the hour mode indicates when the trips occur, and the pickup and dropoff modes indicate where the described trips started and ended. Reading these plots of components in this manner allows us to describe coherent trends in the data.
As latitude and longitude location data is not easily human readable, for selected components, we plot the high-scoring pickup and dropoff locations. The pickup and dropoff locations are color-coded and scaled by their score in the component.
# Custom plotting function for geographic visualization of this taxi decomposition
def taxi_plot(decomp, comp_id, zoom_level=12):
df = pd.DataFrame(
np.array([
np.concatenate([decomp.factors[1][:, comp_id], decomp.factors[2][:, comp_id]]),
np.concatenate([
len(decomp.factors[1][:, comp_id])*['Pickup Location'],
len(decomp.factors[2][:, comp_id])*['Dropoff Location']
]),
np.concatenate([
[l.split('__') for l in decomp.labels[1]], [l.split('__') for l in decomp.labels[2]]
])[:, 0],
np.concatenate([
[l.split('__') for l in decomp.labels[1]], [l.split('__') for l in decomp.labels[2]]
])[:, 1]
]).T,
columns=['score', 'type', 'lon', 'lat']
)
df['score'] = df['score'].astype(float)
df['lat'] = df['lat'].astype(float)
df['lon'] = df['lon'].astype(float)
df = df[df['score'] != 0.0]
fig = px.scatter_mapbox(df, lat="lat", lon="lon", zoom=zoom_level, height=800, size="score", color="type")
fig.update_layout(mapbox_style="open-street-map")
fig.update_layout(margin={"r":0,"t":0,"l":0,"b":0})
fig.show()
Nightlife¶
Component 27 clusters together the taxi rides associated with nightlife. By plotting the scores in the mode vectors of the component, we see that only Friday and Saturday evening hours in the time mode have non-zero scores. Additionally, the high-scoring pickup and dropoff locations are the Meatpacking District, Greenwich Village, and the Lower East Side, all of which are known for their bars and restaurants. The pickup and dropoff locations are plotted on the map below.
plot_component(decomp, 27)
The datetime pattern is especially isolated here, peaking at 1-2am on Saturday and Sunday mornings. Also, we can see a few people starting the weekend early on Thursday night!
The other two modes are better interpreted with the geographic visualization below.
taxi_plot(decomp, 27)
Penn Station Dropoffs¶
People frequently take cabs to Penn Station to catch trains leaving the city, so it is natural that there is a component capturing this behavior. The dominant dropoff location in Component 5 is Penn Station, and the pickup locations are in the surrounding area. These are plotted in the map below. The time mode in the component plots below show that these trips happen daily. Notably, there is a spike Monday through Friday at 6pm. This uptick in Penn Station dropoffs during those hours corresponds to the many commuters leaving the city for the day.
plot_component(decomp, 5)
taxi_plot(decomp, 5)
Rides to an Airport¶
Component 22 consists mainly of dropoffs at JFK, LaGuardia, and Newark airports, while the pickups are mainly in Manhattan. These high-scoring locations are plotted in the map below. As expected, the time mode in the component plots below shows that these trips occurs daily. Note that each day's scores have a bimodal shape. That is, there are morning and afternoon peaks corresponding to when the majority of flights leave.
plot_component(decomp, 22)
taxi_plot(decomp, 22, 10)
Key Takeaways ¶
In this notebook, we used tensor decompositions to find distinct "patterns of life" in entity-based spatiotemporal data. We created a tensor encoding the time, pickup location, and dropoff location of taxi trips in NYC during one week, and we decomposed it to obtain encapsulations of behaviors in the original data. Each component represented one such behavior, ordered by weight, which ranged from rush-hour trips to Penn Station to taxi rides after a night out. Intuitively, by finding a compressed representation of the original tensor, the decomposition clustered similar trips together. Notably, we were able to isolate these coherent patterns from only three features with minimal feature engineering (we only discretized the range of possible values). There is nothing special about the taxi data, and tensor decompositions can summarize the activities found in any entity-based data. An interesting dataset for further exploration is Vessel Traffic Data, which is produced by all passenger vessels by law. It includes, among other features, ship identification, time, and location. After decomposing a tensor encoding these features, one would expect to find distinct components representing the behaviors of different types of ships, such as commerical ships in the Port of New York and New Jersey or recreational vehicles in the Long Island Sound.